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Implementing an adaptive intelligent tutoring system as an instructional supplement 


Blended learning is an instructional model that combines teacher instruction with 
online or digital learning. It can also help to enable more personalized approaches by freeing 
some of the teacher’s time, which otherwise would have been used to provide whole-class 
instruction, so they can focus on individual students while other students are using technology 
(Pane et al., 2017). Some empirical evidence suggests that technology-based curricula can help 
personalize students’ learning experiences and facilitate the development of mathematical 
skills (Koedinger et al., 2000; Ritter et al., 2007; Schacter, 1999; Wenglinsky, 1998). A meta- 
analysis conducted by Means et al. (2010) estimated that interventions combining online and 
face-to-face instruction in a blended-learning approach appeared to produce more positive 
effects than either online or face-to-face instruction alone. Subsequently, more evidence has 
been emerging from rigorous studies on learning systems operating in classroom settings, and 
the evidence of efficacy has been mixed (Morgan & Ritter, 2002; Pane et al., 2010; 2014). 
These studies highlighted some of the challenges implementing the systems in classrooms, 
which may contribute to a lack of consistent positive effects on student learning. 

Literature on blended learning stresses the importance of (1) intentional integration of 
the two modalities to create an integrated learning experience and (2) effective professional 
development on how to divide the instructional role between teacher and technology (Bailey et 
al., 2013; Bowyer & Chambers, 2017; Ferdig & Kennedy, 2014; Patrick et al., 2013; Watson et 
al., 2013). Yet, robust research on blended learning implementation is sparse, with some 
authors calling on future research to develop better ways to measure implementation. Both 
Karam et al. (2016) and Snodgrass, Rangel et al. (2015) identified the need to investigate 
dosage (how much of the intervention is needed to have an effect) and fidelity (whether the 
intervention is implemented as intended) as important aspects of blended learning 


implementation. Karam et al. (2016) extended this by suggesting that, irrespective of 


implementation fidelity, it is important to understand how the intervention changes 
instructional practices. Bowyer and Chambers (2017) and Baily et al. (2013) echo the 
importance of examining teaching, learning, and instructional design. Others discuss the 
importance of learning analytics (data collected by the software) to gain a deeper insight into 
the implementation of digital innovations and their effects on teaching and learning. Learning 
analytics can enable evaluators to examine how a user interacts with various components and 
weigh this along with dosage to gain insight into student engagement with the technology and 
the impact of specific components (Snodgrass Rangel et al., 2015). The present study builds on 
many of these recommendations to help enrich the field’s understanding of blended learning 
implementation. 

Present study 

This article studies the implementation of ALEKS (Assessment and LEarning in 
Knowledge Spaces) in a blended-learning model. ALEKS is an intelligent tutoring system for 
mathematics designed to be integrated with existing curricula. This is the first rigorous 
experimental evaluation of ALEKS in K-12 in-school settings, and the first to gather extensive 
implementation data. We selected ALEKS for this study because it was in widespread use in 
high school classrooms and had shown promise for improving student achievement but was not 
yet subjected to a rigorous test of efficacy for that use. 

Prior research on ALEKS generally relied on non-causal methods or examined its use 
in other contexts. Encouraging effects have been reported for K-12 students in afterschool 
programs; for post-secondary students; and for adult learners (Ahlgren & Harper, 2009; Baxter 
& Thibodeau, 2011; Carpenter & Hanna, 2006; Craig et al., 2014; Hu et al., 2012; Hu et al., 
2013; Huang et al., 2016). Craig et al. (2013) examined the relationship between student 
mindset and engagement with ALEKS. A study by the developers found that the system 
showed promise in improving algebra readiness (ALEKS Corporation, 2011) but did not focus 


on implementation. Two middle school studies by Sullins et al. (2013) found positive 


correlations between usage and student achievement, although it is unclear how online and 
traditional learning were blended in that implementation. 

Our study examined the use of ALEKS as a supplement to an existing high school 
algebra curriculum, as part of a randomized controlled trial studying its efficacy for improving 
achievement. Approximately 2,500 students were randomly assigned to use ALEKS as a 
supplement to the district’s algebra curriculum, or a control group that just used the algebra 
curriculum. The details of the study design and results of the outcomes study will be reported 
in a separate publication. Briefly, the study found no significant effect on an end-of-course 
algebra exam. The focus of this article is to address gaps in the research literature on 
implementing intelligent, adaptive learning systems in a blended learning environment (Bailey 
et al., 2013; Bowyer & Chambers, 2017; Karam et al., 2016; Snodgrass, Rangel et al., 2015). 
As such, it focuses on the classrooms in the experiment that were assigned to use ALEKS. 

The implementation study is guided by the following research questions: (1) What 
models did teachers use to integrate ALEKS into instruction? (2) To what extent did 
implementation adhere to the core aspects of the ALEKS design? and (3) To what extent did 
blended use of ALEKS enable personalized instruction? 

Basic overview of the ALEKS experience for students and teachers 

Upon first entry into the ALEKS system, the student takes a diagnostic (or placement) 
assessment that seeks to uncover what the student already knows or does not know. The result 
of this assessment is displayed to students. Students view the algebra topics already mastered 
and those not yet mastered. ALEKS curates a customized path of “ready to learn” algebra 
topics for the student because they have mastered the necessary prerequisites. The student can 
choose any topic from among the ready-to-learn topics for the ALEKS course. The ALEKS 
algebra course includes more than 350 topics, some of which reflect prerequisite algebra topics. 
This mechanism thus implements a mastery-based approach to progression through the 


software, while still giving students some choices of what to work on. 


Theoretically, this ready-to-learn, mastery-based strategy could be beneficial compared 
to typical classroom instruction where all students cover material at the same pace regardless 
of whether they understand key prerequisites. Using this strategy, students can fill in gaps of 
understanding, work a zone of proximal development (Vygotsky, 1930—1934/1978), experience 
success, and build a better foundation for learning more difficult topics. This can potentially 
lead to more robust learning. However, if students spend a substantial amount of time filling in 
prerequisite gaps, their learning, even if substantial, might not align with the algebra 
curriculum—thus posing a tension known as the mastery versus coverage dilemma (Slavin, 
1987). Implementing ALEKS as a supplement may ease the tension somewhat because teachers 
can cover the required course content and standards while ALEKS provides students with 
personalized instruction to support increased mastery. 

ALEKS personalizes student experiences using an algorithm based on the Knowledge 
Space Theory. The main instructional activity within ALEKS is solving problems in a 
constructed-response environment that uses realistic input tools and avoids multiple-choice 
questions to help ensure that the student demonstrates mastery. The student receives immediate 
feedback and has access to step-by-step explanations (essentially, worked examples) of how to 
solve problems. Mathematical terms and concepts that appear in the problems and explanations 
are hyperlinked to a central glossary. After learning a topic, the student returns to an updated 
ALEKS pie to choose the next topic to learn. Periodic progress assessments are used to 
confirm retention, and if the student no longer demonstrates mastery of a previously-completed 
topic it is returned to the “need to learn” portion of the pie. The student continues to participate 
in this process of learning and assessment until they reach their learning goals. The assumed 
learning goal in ALEKS is 100 % completion of the course, but instructors can set intermediate 
goals for their students. 

For teachers, ALEKS provides a learning management system to support class 


administration, instruction, customizing course content, and progress monitoring. Various 


reports can be used as part of a data-driven, decision-making process to guide their work, both 
during whole-class instruction and for small group or individualized instruction while other 
students are using the software. For example, ALEKS can identify topics that require 
additional focus for the whole class or identify groups of students who are all ready to learn the 
same topics. Teachers can work with these groups or assign them to work collaboratively using 
ALEKS-generated worksheets. Teachers can also use the software to create homework 
assignments and quizzes. 

Methods 
Study Design and Participants 

The study was implemented in a large, urban school district in the mid-Atlantic region, 
where levels of mathematics proficiency measured by the 2014 state algebra I assessment were 
well below state averages, with performance gaps by race/ethnicity. Nine high schools 
participated in the study for two consecutive school years, with different cohorts of algebra 
students each year, but, the same teachers to the extent possible given teacher turnover. The 
two-year implementation allowed for possible improvement in implementation after teachers 
gained experience with ALEKS. The study actually took place over three years because some 
schools entered the study later. 

Table 1 summarizes characteristics of the participating schools. Five offered open 
admission to students coming from middle schools within a feeder region; students from 
outside the feeder region could apply for admission. The other four schools required all students 
to apply, with admission criteria related to attendance, punctuality, behavior, grades, and 
standardized test scores. 

The vast majority of students enrolled in study classes were non-white and about half 
of the students in study classes were female (Table 1). Most students (66%) were economically 
disadvantaged. Fourteen percent of students had an Individualized Education Plan and very 


few students were limited English proficient. The vast majority of students enrolled in study 


Table 1 Characteristics of participating students by study school 


School Participating School Number of | Non-white (%) Female (%) Economic Disability (%) Limited English 
School Years Admissions Students Disadvantage Proficient (%) 
Criteria (%) 

S01 2014-15 Yes 208 99 56 77 16 2 
2015-16 

S02 2014-15 209 100 36 71 36 2 
2015-16 

S03 2014-15 487 85 43 62 7 9 
2015-16 

S04 2014-15 291 100 45 71 14 1 
2015-16 

S05 2014-15 Yes 243 89 58 53 5 3 
2015-16 

S06 2014-15 Yes 214 99 33 71 23 2 
2015-16 

S07 2015-16 354 75 44 66 9 10 
2016-17 

S08 2015-16 Yes 280 80 59 51 8 2 
2016-17 

S09 2015-16 208 99 56 80 24 16 
2016-17 


Overall 2494 90 47 66 14 1 


classes were enrolled in 9th or 10th grade. On the of 8th grade state mathematics assessment— 
the assessment used as a pretest—the average study student scored at about the 23rd percentile 
statewide. 

With one exception, all participating teachers were veteran, with more than 5 years 
experience teaching mathematics. Of the 38 teachers, three had limited exposure to ALEKS 
prior to the study. No teachers implemented personalized learning previously; a few teachers 
occasionally used software or applications to supplement the district curriculum. 

The implementation model in this study involved using ALEKS as a blended learning 
supplement to those existing district materials, including a scope and sequence plan. Teachers 
received the initial training from ALEKS staff—employees of the software developer focused 
on training and supporting implementation—prior to implementing the software. Training 
provided an overview of the software features for students and teachers, and covered guidelines 
on implementation, including specific information on how long and how often students should 
be working in ALEKS. It also provided instruction in how to read ALEKS reports to find key 
indicators of strengths, weaknesses, and progress, and best practices. In the 2015-16 and 2016— 


17 school years, teachers received monthly in-person visits from ALEKS staff. 


Data collection 

The research team conducted three site visits per year to all classes in the study. The 
visits collected information on instruction, student engagement and ALEKS implementation. 
The first site visit occurred in the fall, approximately one month after schools deemed course 
enrollment stable. The third site visit generally occurred in the weeks before the state 
assessment, and the second visit was about halfway between the other two. Each site visit 
included observations of instruction and teacher interviews. 

We designed classroom observation and teacher interview protocols, informed by an 


implementation checklist used by ALEKS staff for their implementation support visits, material 


covered during teacher training sessions, and protocols used in previous studies (e.g. Augustine 
et al., 2016; Karam et al., 2016). The checklist covers several points the company believes are 
important for the software to work effectively in a blended environment, such as ensuring the 
algebra curriculum is appropriate given the student’s preparation, use by students of notebooks, 
and use by teachers of reporting features to monitor progress and make instructional decisions. 
At teacher trainings, ALEKS staff recommended each student use the software for at least two 
hours per week, or at least 60 hours over the school year. Teachers were encouraged to consider 
a variety of models for how they might integrate ALEKS into their course to accomplish usage 
and blended learning guidelines. An online repository contributed to by teachers who had used 
ALEKS previously also offered some possibilities. 

Observation domains included time on task, student engagement, indicators of general 
instructional quality, and indicators of quality for implementation of ALEKS. Wereviewed the 
instrument with ALEKS staff and piloted it in classes that did not participate in the study. 
During the pilot, we refined the protocol and established inter-rater agreement of 100% on 
each item. When two raters did not initially agree, we discussed the item and arrived at 
consensus. Raters continued to observe classes until achieving 100% agreement in an 
observation. After each observation of a study class, one rater reviewed the running record of 
instruction and ratings for agreement. The interview protocol covered curriculum, planning, 
instructional practices, and support received for implementing the algebra I curriculum and 
ALEKS. Interviews were semi-structured and conducted in-person. 

ALEKS provided student-level logs of software usage. In this paper we summarize 
usage time (in hours) for students enrolled in ALEKS classes, and classroom level medians of 
usage. Other, more complex analyses of the log data are planned for other publications. 
Analytic approach 
Given the flexibility offered to teachers in how to use ALEKS, as well as lack of prior research 


on ALEKS implementation, we developed a framework for analyzing implementation (Dane & 


Scheider, 1998; Dusenbury et al., 2003; Carroll et al., 2007) in relation to the three research 
questions, reiterated here: (1) What models did teachers use to integrate ALEKS into 
instruction? (2) To what extent did implementation adhere to the core aspects of the ALEKS 
design? and (3) To what extent did teachers use key features of ALEKS to facilitate 
personalized instruction? We specified these research questions to address the gaps in literature 
on implementation in a blended learning environment, which include adherence to the core 
components of an intervention, dosage, and designs for blending instruction. The research 
questions were also intended to identify the components that may be essential to the 
intervention if ALEKS was found to have a positive effect on student outcomes. Analysis made 
use of data gathered from observations, interviews, and software logs. Themes were identified 
through an iterative process, using Dedoose software to assist with coding the data and 
identifying implementation themes. The analytic approach for each research question follows: 

Implementation models for integrating ALEKS into instruction. We examined 
teacher-developed implementation models and plans for how both teacher and software would 
deliver instruction. We conducted a thematic analysis to identify contextual factors that 
affected enactment of the model. 

Adherence to core aspects of the ALEKS design. Using information from teacher 
trainings, interviews with ALEKS implementation staff, the implementation checklist, and 
software logs, we developed a rubric for capturing four key aspects of the ALEKS design, 
shown in Table 2. We rated adherence to using a3-point Likert scale of no adherence (0), 
partial adherence (1), or full adherence (2) for each of the four key aspects of the ALEKS 
design. There was a total possible score of 8 for adherence to the core aspects of the ALEKS 
design. 

Extent to which ALEKS enabled personalized instruction. As discussed earlier, the 
literature on blended and personalized learning says that teacher and computer-led instruction 


should provide an integrated learning experience. We identified five discrete domains after 


Table 2 Rubric for adherence to core aspects of ALEKS design 


Item 


Student Placement in the 
Correct ALEKS Course 


Use of ALEKS problem- 
solving tools only 


Use of student pathways in 
ALEKS determined by the 
algorithm 


Hours of usage of ALEKS 


Data Sources 
Teacher interviews; classroom 
observations 


Teacher interviews; classroom 
observations 


Teacher interviews 


Software logs 


No adherence (0) 


Few students are placed in the 
correct ALEKS course 


Students almost always used 
tools other than those intended 
by ALEKS 


Overrode the personalized 
student pathway majority of 
year 


0-29 hours 


Partial adherence (1) 


Most students are placed in the 
correct ALEKS course 


Students sometimes used tools 
other than those intended by 
ALEKS 


Overrode the personalized 
student pathway for a portion of 
the year 


30-59 hours 


Full adherence (2) 


All students are placed in the 
correct ALEKS course 


Students used only tools 
provided by ALEKS 


Students used the ALEKS 
personalized student pathway 
exclusively 


60 or more hours 


Table 3 Rubric for extent to which ALEKS enabled personalized instruction 


Domain 
Use of ALEKS Performance 
Data 


Incorporation of ALEKS in 
Course Grade 


Integration of Instructional 
Content 


Differentiated Instructional 
Practices 


Monitoring Student Progress 


Not at all integrated (0 
Teacher did not look at ALEKS data 


ALEKS was not part of the course 
grade 


Students selected ready-to-learn topics 
without guidance from the teacher 


Teacher used the same practices and 
resources as non-ALEKS classes to 
differentiate instruction 


Teacher did not engage when students 
used ALEKS 


Somewhat integrated (1 
Teacher looked at student or class-level 
reports of usage and/or progress 


Student usage hours drove students’ 
ALEKS grade 


Teacher directed students to ready-to- 
learn topics that aligned with teacher-led 
instruction 


Teacher used ALEKS data to create 
small groups or identify standards for 
individual instruction but used the same 
resources as non-ALEKS classes 
Teacher walked around the room; 
directed students to complete problems 
or stay on task; met with students one- 
on-one when they had questions; 
students used ALEKS Notebook 


Integrated (2 

Teacher used student or class-level ALEKS data 
when planning instruction (e.g., shortening or 
skipping instruction where the majority of 
students demonstrated mastery) 


Student growth or progress drove students’ 
ALEKS grade (i.e. mastery-based grade) 


Teacher used customized materials to deliver 
small group or personalized instruction to 
students; teacher prioritizing ready-to-learn 
topics related to classroom instruction 


Teacher used customized ALEKS materials to 
deliver small group or personalized instruction 
to students 


Teacher reviewed reports of active student 
engagement while students used ALEKS; met 
with students one-on-one to assess 
understanding of ALEKS problems; teacher 
reviewed ALEKS notebook 


reviewing ALEKS implementation documents, participating in trainings, and reviewing 
implementation strategies submitted to an online repository by teachers where ALEKS 
provided opportunities to inform teacher-led instruction (Table 3). Opportunities for teachers to 
inform ALEKS instruction were constrained by the fact that most ALEKS instructional 
decisions are shaped by internal software analytics. In each domain, the scale illustrates 
practices that are either not integrated (0), somewhat integrated (1), or integrated (2). Ratings 
for each domain were summed to reach an overall score for personalized instruction, with 
maximum of 10. 
Limitations 

The study relies on classroom observations and teacher interviews, which have a 
variety of limitations. We observed classrooms only three times during the year, offering just a 
glimpse of teachers’ everyday instructional practices. In interviews teachers reported on their 
practices between classroom observations to help fill in the gaps. Although the two sources 
corroborated each other, both are vulnerable to biases. Teachers knew in advance when we 
would appear and may have acted differently those days. Self-report data from interviews can 
suffer from a variety of problems, including poor recall or social desirability bias. Our 


observation rubrics were used for the first time in this study and not previously validated. 


Results 


We describe implementation for each element: (1) Implementation models for 
integrating ALEKS into instruction; (2) Adherence to core aspects of the ALEKS design; and 
(3) Extent to which ALEKS enabled personalized instruction. 
Implementation models for integrating ALEKS into instruction 
Implementation and integration. Table 4 summarizes the variation in implementation by 
identifying three broad implementation models: (1) Integrated—teacher integrated ALEKS into 


instruction, (2) ALEKS— teacher taught using ALEKS only, or (3) Teacher-led—teacher did not 


integrate ALEKS at all. The study anticipated use of an integrated model where teacher delivered 


instruction with ALEKS but 38% of classes used only one modality. In nine (23%) classes, teachers 


Table 4 Implementation models and integration approach, and usage by class 


ied Class ars Integration Approach ie 
1 S01_C01 ALEKS 41 
1 S01_C02 ALEKS 4] 
1 S02_C03  Teacher-led 3 
1 S02_C04 Integrated 2-3 days per week 10 
1 S03_C05 Integrated 2-3 days per week 30 
1 S03_C06 Integrated ah only for several months; 2-3 days per 4 
1 S03_C07 Integrated ALEKS only for several months; 2-3 days per 19 
week 

1 S03_C08 Teacher-led 2 
1 S04_C09 Teacher-led 0 
1 S04_C10 Teacher-led 0 

1 S05_Cl1 Integrated 2-3 days per week 12 
1 S05_C12 Integrated Outside of class 14 
1 S06_C13 ALEKS 26 
1 S06_C14. ALEKS 22 
2 SO1lC15  Teacher-led 9 
2 S01_C16  Teacher-led 2 
2 S09_C17 Integrated 3-4 days per week 6 
2 S09 C18 Integrated 2-3 days per week 1 
2 S07_C19  Teacher-led 0 
2 S07_C20 Integrated 2-3 days per week 0 
2 S02_C21 ALEKS 18 
2 S02_C22 Integrated 2-3 days per week 6 
2 S03_C23 Integrated 2-3 days per week 21 
2 S03_C24 Integrated 2-3 days per week 15 
2 S03_C25  Teacher-led 3 
> S03_C26 Integrated 2-3 days per week for a month; teacher-led 4 

remainder of year 

2 S04_C27  Teacher-led 0 
2 S04_C28 Integrated 2 days per week 17 
2 S05_C29 Integrated 2 days per week 18 
2 S05_C30 Integrated Daily small group teacher-led instruction 75 
2 S06_C31 Integrated 2-3 days per week 24 
2 S06_C32 Integrated 2-3 days per week 21 
2 S08 C33 Integrated When time allowed 11 
2 S08 C34 Integrated 2-3 days per week 17 
3 S09 C35 Integrated 1 day per week 0 
3 S09 C36 Integrated 2 days per week 3 
3 S07_C37 Integrated 2 days per week 5 
3 S07_C38 Integrated 2 days per week 4 
3 S08_C39 ALEKS 79 
3 S08 C40 Integrated When time allowed 12 


S School, C Class 


assigned students to complete the diagnostic assessment in ALEKS but almost never assigned subsequent 
use; instruction focused only on the existing algebra curriculum. At the other end of the spectrum, six 
(15%) classes used ALEKS almost exclusively to deliver instruction. Teachers in five of these six classes 
identified the low levels of readiness for algebra I as a primary reason. According to one teacher, “The 
highest level of mastery [on the ALEKS diagnostic] was 11%. This really shows students are not ready 
for algebra. They need as much time as possible in ALEKS. I’m using [ALEKS] to build up their basic 
skills so that they’1l be at a higher level of readiness [for the algebra curriculum].” Nearly all teachers 
using this model planned to eventually deliver an integrated learning experience but never accomplished 
that goal. 

In the other 25 (63%) classes, teachers implemented an integrated model where 
students received instruction from both the teacher and software. Approaches for implementing 
the integrated model varied. In two of the 25 classes, teachers reported using ALEKS in class 
when time allowed, largely when teacher-led instruction ended early. One teacher assigned 
nearly all ALEKS use to occur outside of class and supported such usage by offering 
students access to computers before school daily and during homeroom twice per week. The 
majority (88%) of teachers using an integrated model assigned ALEKS 2 to 3 days per week in 
class. 

Challenges to integrated usage models. Across all three models, teachers identified 
two key barriers to implementation. First, as illustrated in Table 5, class length ranged from 42 
to 90 min. Teachers felt there was insufficient time to cover the algebra course prior to the state 
end-of-course exam, describing the scope and sequence as aggressive and ambitious for the 
most prepared students in the district. Teachers of classes shorter than one hour (26 of 40) were 
most concerned about instructional time. Second, teachers of the majority of classes (36) 
described students as largely unprepared for algebra. Students reportedly struggled with 
foundational mathematics skills (i.e. decimals, regrouping). Even though teachers sometimes 


incorporated remedial instruction in their algebra lessons, they did not “give up” much 


instructional time to deliver remedial instruction. They viewed allocating class time to ALEKS 


as “giving up” time for algebra instruction. 


Table 5 Instructional minutes by school and study year 


Average length Average Remaining in-class 
of class instructional Recommended ALEKS instructional 
School minutes minutes per week usage per week minutes per week 
S01 45 225 120 105 
S02 90 450 120 330 
S03 54 270 120 150 
S04 47 235 120 115 
S05 47 235 120 115 
S06 89 445 120 325 
S07 90 450 120 330 
45 225 120 105 
S08 80 400 120 280 
S09 42 210 120 90 


Several teachers suggested that an additional math class would provide sufficient time 
to integrate both modes of instruction. A review of implementation strategies submitted to the 
ALEKS repository by teachers revealed that most integrated models relied on additional 
instructional time to implement ALEKS, either by adding a supplemental math class for the 
year or several times per week. 

Adherence to core aspects of the ALEKS design 

According to ALEKS staff, four key aspects of the ALEKS design are important to 
optimize its ability to maximize learning: (a) Placing students in the correct ALEKS course; (b) 
Restricting the use of problem solving tools (e.g. calculators) to those offered within the 
ALEKS system; (c) Allowing the ALEKS algorithm to determine student pathways through 
material; and (d) Sufficient use of ALEKS, or dosage. Analysis of adherence to these 
principles excludes classes that did not use ALEKS and thus reflects observations and 
interviews of 31 classes. Table 6 presents item-level and overall adherence scores for each 
class. We rated adherence to using a 3-point Likert scale of no adherence (0), partial adherence 


(1), or full adherence (2). 


Table 6 Adherence to the core aspects of the ALEKS design by class 


Use of student Meets 
Student placement Useof ALEKS pathways in ALEKS recommended 
in the correct problem-solving determined by the hours of usage 

Class ALEKS course tools only algorithm of ALEKS Total adherence 

S01 _COl 
S01 C02 
S02_C04 
S03_C05 
S03_C06 
S03_C07 
S05_Cll 
S05_C12 
S06_C13 
S06_C14 
S09 C17 
S09 C18 
S07_C20 
S02 C21 
S02_C22 
S03_C23 
S03_C24 
S03_C26 
S04 _C28 
S05_C29 
S05_C30 
S06_C31 
S06_C32 
S08_C33 
S08_C34 
S09_C35 
S09_C36 
S07_C37 
S07_C38 
S08_C39 
S08 C40 


S School, C Class 


NNNNNNNNNNNNNNNNNNNNCTCONNNNNNN NY 
coocooocoocooocococooccoocoocoocoooocoeocqcoo 
NNRFNNNNNNNNNNNNONNNNNNNNNNNYFNN NY 
oNncooocoocococCcCOCcCOCNOOCOCOCCCCCOCOCcCCCOoOOCOCUCcCCcCOCcmcCcCUrRr CLcCOOULr LE 
BAWHHAAAAHAADAAHAAHANAAAAANNAHHHHAUAUN 


N 


Placement in the correct ALEKS course. As a standard practice, ALEKS staff 
monitor diagnostic assessment and progress reports to assess if individual students are placed 
in the appropriate ALEKS course. ALEKS recommends moving a student to a lower-level 
course if diagnostic assessment shows less than 15% mastery or if progress is insufficient. In 
the first study year, ALEKS staff contacted teachers of four classes to discuss moving many of 
their students to the High School Preparation for Algebra course. Two of the four teachers 
agreed. The two who declined believed exposure to the algebra content, even if too difficult, 
would better prepare students for the state assessment. 

ALEKS staff conducted a review of diagnostic assessment, usage, and progress data 
before the second year of the study. They determined the vast majority of students would have 
benefitted by using the High School Preparation course. As a result, ALEKS staff redesigned 
the course sequence for 2015-2016 school year. ALEKS placed all students in the High School 
Preparation course in the first semester and promoted them to the Algebra course once they 
mastered between 50 and 70% of the High School Preparation content. If they did not meet this 
criterion, students were promoted in the second semester. Overall, students in 29 of 31 classes 
(94%) were placed in the correct course. 

Use of ALEKS problem-solving tools only. ALEKS assumes that students only use a 
calculator within the software and only when it is provided by the program, and that otherwise 
students solve problems on their own. In every class, students used a non-ALEKS calculator. 
In many classes, teachers distributed graphing calculators and students opted to use them. In 
interviews, some teachers reported encouraging students to use graphing calculators because 
they were permitted during the state assessment. In classes where teachers did not distribute 
graphing calculators, we observed students using calculators available on Chromebooks or 
smart phones. We also observed individual students using websites to solve ALEKS problems 
in a few classes each year. Students entered the ALEKS problem, received a solution, then 


entered the solution into ALEKS. No classes adhered to this core design aspect. 


Use of ALEKS student pathways. The ALEKS algorithm personalizes students’ 
experiences within the software and generates printable material (e.g., worksheet) for offline 
use. ALEKS staff strongly discouraged teachers from overriding this. Yet, teachers of 3 classes 
reported overriding personalized pathways to assign all students the same problems. Two 
teachers assigned all students the same content for approximately one month. The third teacher 
assigned the same content most of the year. Thus, 90% of classes adhered to personalized 
student pathway designed by ALEKS, two classes partially adhered, and one class did not 
adhere. 

Even though the majority of teachers adhered, many expressed a desire to “override 
the algorithm” so that work in ALEKS would address the topics covered during teacher- led 
instruction. Teachers were concerned about the disconnect between the ready-to-learn topics in 
ALEKS and the current content being covered in the class. If more teachers had the skill to 
circumvent the ALEKS pathway, they might have done so. Adherence to the pathway was 
high, but best described as reluctant. 

Recommended versus actual ALEKS usage. All teachers that adopted models that 
included ALEKS usage (ALEKS-only and integrated) aimed to design an implementation 
approach that met or exceeded the recommended usage. Despite these intentions, actual class- 
level usage was significantly below 60 h. As shown in Table 4, the median level of usage 
exceeded 60 h in only two classes (S08 C39 and SOS_C30). The next highest median usage 
was 41 h. Of the four classes with the highest usage levels, three delivered ALEKS-only 
instruction. In the class meeting recommended usage with an integrated approach, all teacher- 
led instruction occurred individually or in small groups while the rest of the class used 
ALEKS. Most classes using an integrated model had less than 30 median usage hours, a level 
significantly below the recommended 60 h. Table 7 summarizes the number of classes 
adopting each implementation model, and the resulting levels of ALEKS use, calculated as the 


average of eachclass’s median student usage hours. Usage was well below the recommended 


60 h in each model. 


Table 7 Summary of implementation models and average of classroom median usage by model 


Average class 


Model Number of classes median usage hours 
Teacher-led 9 2 

ALEKS 6 38 

Integrated 25 16 


Observations highlighted student engagement as a challenge to achieving 
recommended usage levels. When using computers, the vast majority of students did not 
consistently stay on task. Most students intermittently worked in ALEKS and engaged in off- 
task behaviors, such as conversing with peers, or browsing the Internet. In only 2 of 31 classes 
were the vast majority of students on task across observations. ALEKS distinguishes between 
simply logging in and “active” engagement. Thus, usage time reflects students’ active ALEKS 
use, and can depart from teachers’ allocation of instructional time. 

Teachers also identified student attendance as a barrier to robust usage. Most teachers 
reported that a core group of students regularly attended class and the remainder attended 
erratically. The size of the core group of students ranged from very small to approximately half 
of the class. Only one school reported that a majority of students regularly attended. Students 
with irregular attendance may have missed multiple in-class opportunities to use ALEKS and, 
according to teachers, were unlikely to use ALEKS outside of class. Observations aligned with 
teacher reports of low attendance. 

Overall, there was mixed adherence to the four core design aspects of ALEKS (Table 
6). In the vast majority of classes (94%) students were placed in the correct ALEKS course and 
progressed according to their ready-to-learn pathway. Adherence was very low in students’ use 
of ALEKS-only tools to solve problems and median usage hours. As shown in Table 8, total 
adherence scores were the same for both ALEKS-only and integrated models, but there was 
some variation in adherence of components. Classes with an integrated model had low levels 


of usage while classes with an ALEKS-only model had partial adherence to usage levels. 


Table 8 Average adherence to the core aspects of the ALEKS design by domain and model 


ALEKS 


Use of 

Student student 

placement Use of pathways Meets 

in the ALEKS in ALEKS recommended 

correct problem- determined hours of Total 
Number of ALEKS solving by the usage of average 

Model classes course tools only algorithm ALEKS adherence 

6 1 0 2 1 4 
26 2 0 2 0 4 


Integrated 


Extent to which ALEKS enabled personalized instruction 


Our rubric provided ratings for ALEKS classes in five domains derived from ALEKS documents 
and trainings that related to how integrated instruction can enable personalized learning (Table 9). This 
analysis included all classes using ALEKS (n = 31). 

Using ALEKS performance data. All teachers reported knowledge of the data available in 
ALEKS and received training in interpreting it. Yet, nearly 30% of teachers whose classes used ALEKS 
did not review available student or class-level data. Most teachers (59%) did review ALEKS reports at 
least weekly to assess usage hours, but only 4 (13%) considered ALEKS data when planning their 
teacher-led instruction, a necessity to get a rating of fully integrated. One of those 4 teachers used data to 
identify topics that did not require teacher-class instruction, and 3 of them used ALEKS data to create 
small groups for teacher-led instruction. 

Incorporating ALEKS data into course grade. Twenty-nine of 31 teachers incorporated 
ALEKS data into students’ grades, though ALEKS’s weighting was not high (10% to 20%) in most 
classes. Some classes gave extra credit for ALEKS usage and qualified as somewhat integrated. All of the 
teachers who incorporated ALEKS in students’ course grades reported using the Time and Topic report 
as the primary data source for grading, which implied that usage hours drove the ALEKS portion of 
grades. No classes met a rating of fully integrated, which required use of mastery and mastery rate goals 
(e.g., mastered topics per week). 

Integrating instructional content. This domain considered the extent to which computer and 
teacher-led instruction informed each other. Ready-to-learn topics in ALEKS were personalized to 
students, whereas standards for instruction in the existing curriculum were course wide. In all but two 
classes, teachers did not integrate the two. Teachers described ALEKS and teacher- led instruction as 


independent. 


Table 9 Extent to which ALEKS enabled personalized instruction by domain and class 


Use of student Incorporationof Alignmentof Differentiated Monitoring Total 
performance ALEKS in instructional instructional student average 
Class data student grade content practices progress integration 
S01 Col 1 1 0 0 2 4 
SO1_CO2 1 1 0 0 1 3 
S02 C04 0 1 0 0 1 2 
S03_CO5 1 1 0 0 1 3 
S03_C06 0 1 0 0 1 2 
S03_C07 0 1 0 0 1 2 
S05 Cll 2 1 1 2 2 8 
S05_C12 0 1 0 0 0 1 
S06 C13 1 1 0 0 1 3 
S06 C14 1 1 0 0 1 3 
S09 C17 1 1 0 0 1 3 
S09_C18 0 1 0 0 0 1 
S07_C20 1 1 0 0 1 3 
S02._C21 0 1 0 0 0 1 
S02.C22 1 1 0 0 1 3 
S03_C23 1 1 0 0 1 3 
S03_C24 1 1 0 0 1 3 
S03_C26 0 0 0 0 0 0 
S04 C28 0 1 0 0 0 1 
S05_C29 1 1 0 0 1 3 
S05_C30 2 1 0 2 2 8 
S06 C31 1 1 0 0 1 3 
S06 C32 1 1 0 0 1 3 
S08 C33 0 0 0 0 0 0 
S08 C34 1 1 0 0 1 3 
S09 C35 2 1 0 0 2 5 
S09 C36 2 1 0 0 1 4 
S07_C37 1 1 0 0 1 3 
S07_C38 1 1 0 0 1 3 
S08_C39_ 1 1 0 0 1 3 
S08 C40 1 1 0 0 0 2 


S School, C Class 


One way for teachers to integrate modalities is to direct students to a list of ready-to- 
learn topics that are “relevant” to standards being covered during teacher-led instruction. We 
observed only one teacher who regularly did this. According to the teacher, “Students would 
[experience] better growth if they worked on topics that supported standards covered in class 
and homework. They would get more out of both [instructional modalities]. Students could 
choose topics, but the teacher aimed to align content from both instructional modalities. 

At the end of each year, the majority of teachers requested more support in integrating 
ALEKS with the algebra scope and sequence. According to teachers, the disparity between 
each student’s ready-to-learn topics and algebra standards was too vast. Teachers knew what 
standards they addressed during their instruction and could see the ready- to-learn topics, but 
mapping between the two was challenging. A few teachers suggested this was difficult 
because they did not know which ready-to-learn topics would most closely align with the 
algebra standards covered in class. 

Differentiating instructional practices. ALEKS generated personalized materials on 
demand that teachers could use when delivering small group or individualized instruction to 
students. The majority of teachers (94%) did not use these resources. The two teachers who 
used them did so while the rest of the class used ALEKS, as intended. 

Monitoring student progress. Most teachers (78%) monitored progress as students 
used ALEKS. Twenty of the 24 teachers who did this used a somewhat integrated approach, 
walking around the classroom and periodically observing students’ screens as they solved 
problems. A small group of teachers, 4 of 24, implemented an approach rated fully integrated 
by conducting one-on-one meetings with students to assess students’ understanding of the 
content. ALEKS intended for students to have a dedicated notebook to solve the program’s 
open-ended free response questions and for teachers to use as a tool to help monitor student 
progress. Though ALEKS staff distributed notebooks, they were rarely in evidence in our 


observations and teachers rarely reported using them. 


In summary, the overall extent to which teachers enacted personalized instruction with 
support from ALEKS was low. The total possible score was 10 and the median score was 3. 
Two teachers received a score of 8, meaning they somewhat or fully integrated ALEKS with 


their instruction in each domain. These ratings did not vary across models (Table 10). 


Table 10 Average extent to which ALEKS enabled personalized instruction by domain and model 


Use of Alignment 
Number _ student Incorporation of Differentiated Monitoring Total 
of performance of ALEKSin instructional instructional student average 
Model classes _ Data student grade__content practices progress integration 
ALEKS 6 1 1 0 0 1 3 
Integrated 26 1 1 0 0 1 3 
Discussion 


In this final section, we discuss implementation factors specific to this study that may present 
challenges to other implementations of technology in a blended, personalized model, and 
considerations for future research. 
Implementation factors 

Of the 31 classes that used ALEKS, only one class met recommended usage levels and 
demonstrated high levels of personalized instruction. There was insufficient dosage and very 
little personalization of instruction in the vast majority of classes that used ALEKS. Teachers 
identified a number of implementation challenges for both dosage and personalization. These 
factors hindered overall implementation and were consistent with prior implementation 
research (Ely, 1990, 1999; Fixsen et al., 2005; Damschroder et al., 2009; Bertram et al., 2015). 

Teachers in the majority (36 of 40) of classes described students as largely unprepared 
for algebra and in need of extensive remedial instruction. On the 8th grade state mathematics 
assessment—the assessment used as a pretest—the average study student was 0.75 standard 
deviations below the state mean, or at about the 23rd percentile statewide. By and large 


teachers reported ALEKS met a “critical need” because it delivered remedial instruction in 


foundational skills the standard algebra curriculum did not and described the curriculum, 
which did not address gaps in foundational mathematics, as unsatisfactory. There was strong 
buy-in and support for ALEKS, critical factors for successful implementation (Damschroder et 
al. 2009; Ely 1990, 1999), because it could improve students’ performance in mathematics 
(Datnow & Stringfield, 2000; Newmann et al., 2001; Honig & Hatch, 2004). 

Yet, the majority of teachers expressed a tension in allocating instructional time to 
the use of ALEKS at the recommended levels. Teachers felt there was insufficient time to 
cover the algebra course prior to the state end of course algebra assessment, describing the 
scope and sequence as aggressive and ambitious for the most prepared students in the district. 
Even though teachers incorporated remedial instruction in their algebra lessons as a standard 
practice, they did not perceive themselves as “giving up” much instructional time. In contrast, 
ALEKS required 2 h per week, which was nearly half of the instructional time for teachers of 
classes with less than 5 h per week of instructional time (26 of 40 classes). Teachers 
reported a tension between remediation and simultaneous preparation for the high-stakes 
algebra test. Ultimately, teachers were responsible for students’ performance on the end of 
course assessment. Student performance on the algebra assessment was not tied to graduation 
requirements but it did factor into the accountability scores for high schools. The relative 
advantage (Rogers, 1995; Damschroder et al., 2009) for “exposure to” and “practice with” the 
algebra content on the end of course assessment was described as greater than remedial 
instruction. Algebra instruction was more aligned with school goals, which resulted in 
teachers allocating less time to ALEKS (Bryk & Schneider, 2003; Roehrig & Kruse, 2005; 
Honig & Hatch, 2004). Teachers identified an overall need for more instructional time, 
specifically a second mathematics class, in order to achieve the goals of filling in knowledge 
gaps and preparing students for the end of course assessment. 

Teacher readiness to implement a new practice was also a key implementation factor. 


A few teachers had prior experience using online resources (e.g., PLATO, Khan Academy) to 


supplement instruction. No teachers had prior experience implementing personalized 
learning by integrating technology into instruction. In interviews, many teachers 
acknowledged education was moving in the direction of blended and personalized models; 
however, they were not early adopters of the innovative approach. Seven teachers participated 
in the study for the intended two years. Thus, the majority of teachers had no opportunity to 
apply lessons learned during their first year of implementation in a subsequent year. We did 
not analyze changes in implementation from the first to second year of implementation because 
of the small sample size. The National Implementation Research Network implementation 
model suggests changes in practice to integrate an innovation take up to 5 years, with initial 
implementation taking approximately 3 years (Fixsen et al., 2013). According to this model, 
all teachers were in early phases of skill development and changing their practices and it is 
unlikely teachers would exhibit strong implementation (Ensminger et al., 2004; Ebersole & 
Vordan, 2003; Rogers, 2000). 

Further, the initial implementation stage is particularly challenging and requires 
considerable support from coaches (Joyce & Showers, 2002), as well as broader systems 
supports (Aladjem & Borman, 2006; Schofield, 2004). Throughout the study, teachers 
identified integrating ALEKS with the district algebra curriculum as an ongoing challenge. 
Nearly all teachers spoke positively about the support they received through regular in-person 
visits, emails, and text messages from ALEKS staff. However, they described this support as 
primarily technical (e.g., creating reports, navigating the teacher interface) and their mastery of 
ALEKS as basic. According to teachers, the resources ALEKS staff directed teachers to for 
implementation strategies provided very limited information on designing models or 
integrating ALEKS with the regular mathematics curriculum. Support from ALEKS staff did 
not extend to designing and implementing a blended model with the standard curriculum. 
Similarly, teachers reported almost no support from school or district staff on how to design 


and implement a blended, personalized model. Teachers also did not receive instruction in how 


to use the Chromebooks provided by the study to support implementation; some teachers 
reported struggling to master the technology. Overall, teachers’ knowledge and skills with 
ALEKS, personalized learning, and, in some cases, technology were a barrier to 
implementation (Hew & Brush, 2007; Ensminger et al., 2004; Ebersole & Vordan, 2003; 
Damschroder et al., 2009; Ely, 1990, 1999). 

Several other factors emerged as barriers to strong implementation in participating 
classes. Many teachers reported that less than half of their students attended class regularly. In 
addition, the vast majority of students were off task during observations of both ALEKS and 
standard instruction. This, along with poor attendance, hindered dosage, which the ALEKS 
software only counts when students actively engage with the software. Because of these 
factors, most students had reported usage lower than the amount of time teachers made 
available for ALEKS use. 

Four participating schools had selective admissions criteria and five were open 
admission. The class achieving dosage and greater extent of personalized instruction was at the 
school with the most stringent admission criteria. The two other classes with promising 
implementation (dosage or extent of personalization) were also at schools with selective 
admissions. Although this did not guarantee better overall implementation, selective schools 
were more likely to have higher attendance rates and algebra proficiency results. 
Suggestions for implementing and studying technology-enabled personalization 

Measured by dosage and quality, implementation of robust technology-enabled 
personalized learning was very limited in this study, and there was no effect on student 
achievement. It is clear that further studies with stronger implementation and dosage, and 
careful documentation of implementation, may be needed to understand whether and how 
systems like this can support student learning. Our study also offers some more general 


recommendations that may improve implementation of any technology intended to enable 


blended, personalized instruction, and for researchers studying these efforts. 

Consider teacher buy-in and school support. Implementing new instructional 
models is challenging and more likely to be successful with strong support and buy-in at 
multiple levels (Ebersole & Vordan, 2003; Ensminger et al., 2004). It is also important to give 
teachers time to pilot interventions and experiment with implementation of technology-enabled 
personalized learning (Brenner & Brill, 2016). Further, consider whether there is support for 
implementing personalized learning from school (e.g., principal, department chair) and district 
leaders, as well as the developers (Boddy & Macbeth, 2000; Scott, 2000). Coaching and 
support are important during initial implementation of innovations and changing long-standing 
professional practices (Hew & Brush, 2007). 

Clarify key components of the intended implementation. We articulated key 
components of ALEKS, which can be refined through further research. In blended learning 
research more broadly, researchers can engage with software and curriculum developers to 
identify key components of software. In this study, if restricting students to use only ALEKS- 
provided tools is central to ALEKS’s design, teachers need to understand this and formulate an 
enforcement plan, while still giving students adequate practice with the tools that will be 
available to them during standardized tests (Hew & Brush, 2007). Communicating these core 
components and developing plans to adhere to them may facilitate greater implementation 
fidelity (Damschroder et al., 2009; Scott, 2000; Surry & Ely, 2002). 

Provide explicit, feasible models for personalization using the blended approach. 
Teachers need more ongoing support to truly integrate online and teacher-led instruction so 
that they complement each other. For example, it may be helpful to provide semi-prescribed 
models, schedules of integration, or a map linking software topics to standards to help teachers 
connect modalities. Strong models for personalized instruction, particularly in classes with a 
large gap in readiness for the course, could be particularly important (Surry & Ely, 2002; 


Boddy & Macbeth, 2000; Malbert et al., 2003). Professional development on how to deliver 


small group instruction in a secondary school setting or use online data to inform instruction 
may be useful. Considering different ways to structure or extend class time may also be helpful 
(Hew & Brush, 2007). Further, researchers, developers and district instructional leaders can 
develop rubrics for assessing the extent to which instructional modalities are integrated to 
provide personalization. 

Consider student engagement more broadly. Student engagement was a challenge in 
classes where students used ALEKS, classes that used only the existing curricula, and classes 
using both. Personalized instruction and technology-based instruction may solve some student 
engagement challenges, but there are also risks that students will use technology as an 
opportunity to go off-task, or that they will not find the software engaging. Identifying and 
implementing strategies that improve engagement across modalities may enhance 


improvements in student achievement. 


Conclusion 


This paper proposes considerations for improving implementation of, and research on, 
adaptive learning software as a supplement to existing curriculum. Our findings identify 
multiple challenges to designing strong implementation models, as well as supports that may 
promote adherence to the core components of software and enable high quality blended 
instruction. Future research needs to explore the extent and shape of implementation with 
different adaptive and personalized systems and to assess whether and how such personalized 


instruction leads to better outcomes. 
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